[Oracle] Data Guard CPU and PSU Patch Installation Detailed Tutorial

For patch installation tutorials for non-Data Guard, please refer to [Oracle] CPU/PSU patch installation detailed tutorial. Data Guard requires Primary and Standby to be patched at the same time, so the steps are more complicated. The main steps are as follows:
1. Stop the log transfer service at Primary;
2. Close the Standby database, patch the software of Standby (note: there is no need to patch the Standby database), start standby in mount state, and do not enable managed recovery;;
3. Close Primary and patch the software and database of Primary;
4. Start Primary database and restart log transmission service;
5. Start Redo Apply on Standby, so that the patch script on Primary will automatically synchronize to Standby;
6. Check that both Primary and Standby are patched.
Here is a concrete example:
1. Stop the log transfer service on Primary

sys@EPAY>select database_role from v$database;
DATABASE_ROLE
----------------
PRIMARY
sys@EPAY>show parameter log_archive_dest_3
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_3                   string      SERVICE=sta ASYNC VALID_FOR=(O
                                                 NLINE_LOGFILES,PRIMARY_ROLE) D
                                                 B_UNIQUE_NAME=epaybk
log_archive_dest_30                  string
log_archive_dest_31                  string
sys@EPAY>alter system set log_archive_dest_state_3=defer scope=both;
System altered.

2. Patch the Oracle software of Standby
2.1 Close database instances, listener, ASM instances, and so on
2.2 Check the version of opatch. If it is not enough, download the latest version
2.3 Patch Oracle software for Standby
2.4 Start Standby to mount state, start listener
(Note: Standby does not need to patch the database itself)

3. Patch Primary
3.1 Close database instances, listener, ASM instances, and so on
3.2 Check the version of opatch. If it is not enough, download the latest version
3.3 Patch Oracle software for Primary
3.4 Patch the Primary database itself

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

4. Start the log transfer service on Primary
4.1 Start Primary listener, database instance, and so on
4.2 Forced registration of services to listener

sys@EPAY>alter system register;
System altered.

4.3 Restart the log transfer service

sys@EPAY>alter system set log_archive_dest_state_3=enable scope=both;
System altered.

Note: To start log transfer, the following errors may occur in alert:

------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE, 
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------

According to the error message, the main library should modify the sys password when making CPU patch, and replace the password file of the standby library with the password file of the main library to solve the error.
5. Standby Start Redo Apply
5.1 open Standby database
5.2 Enable Redo Apply

sys@EPAY>alter database recover managed standby database disconnect from session;
Database altered.

5.3 Verify that Primary and Standby are synchronized
Query the current largest archive log sequence number on the primary side:

sys@EPAY>select max(sequence#) from v$archived_log;
MAX(SEQUENCE#)
--------------
           159

Query the sent archive log on standby side:

sys@EPAY>select sequence#, applied from v$archived_log;

5.4 It can be seen from alert. log that 3 log files are synchronized (i.e. the patch made in primary is synchronized to standby)

alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (epay)
Wed Jul 10 06:03:48 2013
MRP0 started with pid=29, OS id=15030 
MRP0: Background Managed Standby Recovery process started (epay)
 started logmerger process
Wed Jul 10 06:03:53 2013
Managed Standby Recovery not using Real Time Apply
Wed Jul 10 06:04:01 2013
Parallel Media Recovery started with 32 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Wed Jul 10 06:04:01 2013
Completed: alter database recover managed standby database disconnect from session
Media Recovery Log /data/oradata/epay/archivelog/1_157_814716635.dbf
Media Recovery Log /data/oradata/epay/archivelog/1_158_814716635.dbf
Media Recovery Log /data/oradata/epay/archivelog/1_159_814716635.dbf
Media Recovery Waiting for thread 1 sequence 160 (in transit)

6. Check whether the patch was installed successfully later
6.1 Point to opatch lsinventory in primary and standby respectively
6.2 Check whether the patch was installed successfully in the database